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Prologue:  GAO  Recommends  Change  To  Tech/Prod 
Development  Practices  -  Circa  1999 


•  Adopt  a  disciplined,  knowledge-based  method,  evolutionary 
acquisition  process. 

•  Set  standards — such  asTRLs-for  assessing  technology 
readiness  that  are  based  on  a  technology’s  demonstrated 
readiness  for  integration  and  its  criticality  to  the  weapon 
system. 

•  Require  that  technologies  demonstrate  a  high  readiness  level 
-TRL  1-  before  Milestone  B  of  a  major  acquisition. 

•  Provide  more  flexibility  to  acquisition  programs  with  regard  to 
a  weapon  system’s  performance  requirements. 

•  Consider  requiring  S&T  organizations  to  mature  technologies 
further  and  empower  them  with  additional  funding  and 
improved  organization. 
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Outcomes  Since  Then: 

Outcomes  From  DOD  Development  Efforts  Circa  2005 


•  While  future  demographics  will  demand  more  from  DOD’s 
investment  accounts,  current  practices  yield  systems  at 
unexpectedly  high  cost,  diminished  results 


$  billions 


$89.9 

Current  Plan 
$47.9 

Original  Plan 


■  FY 1998  Plan  ■  FY  2005  Plan 


Source:  Selected  Acquisition  Report  data  (12/31/96  and  12/31/03)  on  the  8  weapon  systems  among  the  highest  R&D  budget  requests  for 
FY  2003.  Note:  All  dollars  are  in  constant  FY  2005  dollars. 
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Outcomes  Since  Then: 

Immature  Technologies  Impact  SDD  Circa  2006 


•  Average  Program  Research,  Development,  Test  and 
Evaluation  Cost  Growth  From  First  Full  Estimate 


Source;  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs.  GAO-06-391.  Washington,  DC.:  March  2006. 
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Outcomes  Since  Then: 

Cost  Growth  on  27  Current  Programs  Circa  2007 


6 


Latest  Review: 

Objective,  Scope,  and  Methodology 


GAO 

Accountability *  *  Integrity  *  Reliability 


•  Objective  was  to  identify  best  ways  to  transition  technologies 
to  product  lines,  assess  practices  used  by  the  military 
services,  and  determine  potential  improvements  for  DOD. 

•  Visited  4  best  practice  companies  (IBM,  Motorola,  3M,  and 
Boeing),  met  with  key  science  and  technology  (S&T)  and 
product  development  officials  to  identify  best  practices  for 
technology  transition  and  supporting  tools. 

•  Met  with  DOD  S&T  officials  from  OSD,  services’  acquisition 
programs,  and  services’  labs.  Also  reviewed  several  lab  & 
acquisition  programs. 

•  Conducted  our  review  from  July  2005  through  May  2006  in 
accordance  with  generally  accepted  government  auditing 
standards. 


7 


Private  Industry  Findings: 

Key  Enablers 
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•  Have  strong  strategic  planning  to  prioritize  technology 
needs  and  a  structured  technology  development 
process  as  precursor  to  transition. 

•  Merge  technology  development  and  product 
development  activities  prior  to  product  launch. 

•  Use  3  tools  to  support  technology  transition: 

•  Relationship  managers 

•  Technology  Transition  Agreements 

•  Metrics 
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Private  Industry  Findings: 

Strategic  Processes  Smooth  Path  To  Transition 

•  Strong  strategic  planning  processes  used  to  identify  and 
react  to  market  needs  quickly. 

•  Structured,  gated  technology  development  process. 


Explore 

Technology  ideas  and  concepts 
are  being  explored 


Develop 

Technology  development  activities 
are  underway 


Technology  transition 

Technology  is  ready  to  transition 
from  lab  to  product  line  team 


Review 


Review 


Deliverables 


Deliverables 


Deliverables 


•  Technology  is  consistent  with  overall 
business  strategy 

•  Technology  is  promising  and  is  likely  to 
meet  needs  for  potential  product  lines 

•  Lab  identifies  potential  products  where 
technology  can  be  used 

•  Key  cost,  benefit,  risk,  marketing, 
manufacturing,  and  life  cycle  manage¬ 
ment  issues  are  identified 

•  Scalability  approaches  are  identified 

•  Technologies  considered  to  be 
intellectual  property  are  identified 


.  Technology  is  consistent  with  techno¬ 
logy  strategy  and  other  relevant 
strategies 

•  Labs  have  high  degree  of  confidence 
the  technology  will  work 

•  Product  line  team  agrees  that  the 
technology  will  meet  its  needs 

•  Technical  requirements  are  identified 

•  Cost,  benefits,  and  risks  are  quantified 

•  Scalability  approach  is  selected 

•  Strategies  for  addressing  intellectual 
property  rights  are  selected 


.  Technology  project  complies  with 
technology  strategy 

•  Technology  is  sound 

•  Technology  meets  product  require¬ 
ments 

•  Cost,  benefit,  and  risks  are  well 
understood 

•  Technology  can  be  scaled  to  a 
magnitude  appropriate  for  practical 
application 

•  Product  line  team  agrees  technology  is 
ready 

•  Intellectual  property  rights  methods 
have  been  pursued 

•  Technology  is  demonstrated  in  an 
operational  environment 

•  Technical  documentation  is  ready  to  be 
given  to  product  line  team 
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Private  Industry  Findings: 

Hybrid  Phase  Merges  Tech  and  Product  Development 


General  Flow  of  Process  Leading  Up  to  Technology  Transition 


Sou  ce:  GAO  analysis  and  pceontaUcn  ot  Icotlng  ctmparies'  praittoaa. 

•  Hybrid  phase  used  to  merge  technology  development  and 
product  development  activities  prior  to  product  launch. 


•  Responsibilities  for  managing  and  funding  technology 
development  gradually  shift  from  labs  to  product  line  during 
this  phase. 
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Private  Industry  Findings: 

Relationship  Managers  Provide  Communications  Link 


Relationship  managers  from  labs  and  product  lines  serve 
as  a  communication  link  between  the  two  communities 
and  work  out  transition  issues. 


LAB  COMMUNITY 


Executive  managers 

1 

Liaisons 

1 

Technologists 


4 


4 


4 


S:<jrce:  Motorola;  GAD  (analyMa  ard  rfes^n^fl,::r|!1- 


* 


PRODUCT  LINE 
COMMUNITY 


Executive  managers 

I 

Liaisons 


Technologists 
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Private  Industry  Findings: 

Technology  Transition  Agreements/Metrics 


•  Technology  transition  agreements  document  decisions 
made  between  labs  ana  product  lines: 

•  Contain  specific  quantifiable  cost,  schedule,  performance,  and 
manufacturability  metrics  the  labs  must  demonstrate  before 
product  line  acceptance. 

•  Feasibility,  relevancy,  and  application  of  each  technology  are 
assessed  in  an  effort  to  identify  potential  barriers  to  technology 
transition. 

•  Identify  lab  and  product  line  funding  commitments,  including 
recurring  manufacturing  costs  associated  with  integrating  the 
new  technology  on  a  product. 

•  Some  agreements  may  include  loaning  key  lab  technologists  to 
the  product  line  for  a  period  of  time  in  order  to  maintain 
momentum  once  a  technology  has  transitioned  to  product  line 
management. 
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DOD  Findings: 

More  Effort  Needed 
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•  DOD  does  not  adequately  prioritize  the  technologies  that 
are  most  critical  to  acquisition  programs. 

•  DOD  does  not  merge  S&T  and  product  line  activities 
prior  to  product  launch;  Transition  often  occurs  at 
product  launch  irrespective  of  whether  technologies  are 
mature. 

•  New  tools  to  support  transition  are  being  used,  but: 

•  Not  as  comprehensive  as  industry  best  practices. 

•  Use  is  not  widespread. 
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DOD  Findings: 
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Inconsistent  Prioritization  and  Dev  Processes 


DOD  is  not  well  positioned  to  develop  and  mature  needed 
technologies  on  time. 

•  Strategic  planning  process  does  not  consistently 
prioritize  technologies  most  critical  to  acquisition 
programs. 

•  Military  services  have  established  S&T  boards  to 
select  and  oversee  new  technology  projects,  which 
increases  visibility  for  some  technologies,  but  the 
scope  varies  across  military  services. 
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DOD  Findings:  Technology  and  Product 
Development  Still  not  Effectively  Aligned 
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DOD  does  not  have  a  structured,  gated  S&T  technology 
development  process  with  deliverables  to  guide  investments. 


S&T  and  acquisition  communities  do  not  communicate  well  and 
are  not  aligned  in  a  way  to  effectively  meet  priorities,  resulting  in: 

•  Irrelevant  technologies  advancing  to  final  stages  of  lab  development  without 
commitment  to  field  the  technologies. 

•  Technology  not  being  ready  to  transition  when  needed. 

•  Acquisition  not  being  prepared  to  take  over  funding  responsibilities. 
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DOD  Findings: 

Technology  Transition  Tools  Underutilized 
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•  Relationship  Managers 

•  Generally  used  to  market  lab  technology;  not  as  a  communication 
tool  to  assist  in  technology  transition. 

•  Technology  Transition  Agreements 

•  Use  and  coverage  vary  greatly  among  service  S&T  programs 

•  Contain  some  of  the  same  elements  seen  in  industry,  but  typically 
do  not  require  the  technology  developer  to  demonstrate  cost 
metrics. 

•  Tool  used  mainly  by  labs;  not  highly  valued  by  acquisition 
community. 

•  Metrics 

•  Few  metrics  used  to  gauge  the  impact  of  investments  or  the 
effectiveness  of  processes  used  to  develop  and  transition 
technologies. 
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DOD  Findings: 

Some  Promising  Initiatives  For  Transition 


•  Advanced  Concept/Joint  Concept  Technology  Demonstration 

(ACTD/JCTD)  --  Goal  is  to  get  technologies  that  meet  critical  needs 
to  users  faster  and  at  lower  cost,  refine  the  the  selection  process  to 
better  match  user  priorities,  and  provide  more  funding  in  early 
stages  of  demonstration. 

•  Manufacturing  Technology  Program  --  Aimed  at  quickly 
identifying  and  solving  technology  transition  problems;  focusing  on 
affordable,  low-risk  development  and  production 

•  Foreign  Comparative  Testing  &  Technology  Transition  Initiative 

--  FCT  identifies,  evaluates,  and  procures  technologies  developed 
by  other  countries.  TTI  speeds  transition  of  DOD  lab  developed 
technologies  to  acquisition  programs. 
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Recommendations 
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•  Develop  gated  process  that  establishes  a  transition  phase  and 
defines  activities  that  should  occur  during  this  phase. 

•  Allocate  more  6.4  funds  for  S&T  to  manage  technologies  to  higher 
readiness  levels  before  they  move  to  acquisition  programs. 

•  Expand  the  use  of  technology  transition  agreements  DOD-wide. 

•  Include  additional  metrics  in  technology  transition  agreements. 

•  Expand  the  use  of  relationship  managers  and  define 
responsibilities. 

•  Adopt  additional  process-oriented  metrics  to  measure  the 
effectiveness  of  S&T  processes  and  the  impact  of  S&T 
investments. 
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For  further  details,  see  GAO  report:  GAO-06-883,  Best 
Practices:  Stronger  Practices  Needed  to  Improve  POD 

Technology  Transition  Processes  (September  2006)  at 
www.qao.gov.  or  contact  Michael  Sullivan  at 
sullivanm@gao.gov. 
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Context:  The  Fiscal  Environment 
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FY  ’05:  $89.95  billion  total  RDT&E  $ 


FY  1998  plan  for  completing 
development  of  8  programs 


Additional  investment 
needed  under  FY  2005 
plan  for  completing  the 
8  programs 


Source:  Selected  Acquisition  Report  data  (12/31/96  and  12/31/03)  on  the  8  weapon  systems  among  the  highest  R&D  budget 
requests  for  FY  2003.  Note:  All  dollars  are  in  constant  FY  2005  dollars. 
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Current  Condition:  TRLS  for  Technologies  At 
The  Time  They  Were  Included  In  Product  Designs 


Non-penetrating  periscope 
Ford’s  adaptive  cruise  control 
Ford’s  night  vision 
Ford’s  voice  activated  controls 
Hughes’  solar  cell  array 
High-speed  planing  craft 
High  power  water  jet 
Weapon  ejection  system 
Diesel  engine 
Helicopter  rotor 
Lightweight  composite  armor 
Helicopter  engine 
Moving  map  and  navigation 
Oxygen  iodine  laser 
Beam  control  system 
IHelmet  mounted  display 
Forward  looking  infrared 
Integrated  avionics 
Data  processors 
Inertial  measurement  unit 
Submunition  warhead 
Infrared  seeker 
Acoustic  targeting 


21 


i 


GAO 

Accountability  *  Integrity  *  Reliability 


Prologue:  Outcomes  of  Past  Major  Weapon 
System  Acquisitions 


Product  development  and 
Associated  technologies 

TRL  at 
program 
launch 

Product  development 

Cost  growth 

Schedule  slippage 

Comanche  helicopter 

101  percent3 

120  percent3 

Engine 

5 

Rotor 

5 

Forward  looking  infrared 

3 

Helmet  mounted  display 

3 

Integrated  avionics 

3 

Brilliant  Anti  Armor  Submunition 

88  percent 

62  percent 

Acoustic  sensor 

2 

Infrared  seeker 

3 

Warhead 

3 

Inertial  measurement  unit 

3 

Data  processors 

3 

Hughes  HS-702  satellite 

None 

None 

Solar  cell  array 

6 

Ford  Jaguar 

None 

None 

Adaptive  cruise  control 

8 

Voice  activated  controls 

8 

aThe  Comanche,  in  particular,  has  experienced  a  great  deal  of  cost  growth  and  schedule  slippage  for  many  reasons,  of 
which  technology  immaturity  is  only  one.  Other  factors,  such  as  changing  the  scope,  funding,  and  pace  of  the  program  for 
affordability  reasons,  have  also  contributed. 
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Outcomes  Since  Then: 

Constructive  Policy  Changes,  Implementation  Challenges 


•  DOD  5000  policy  says  most  of  the  right  things  about 

separating  technology  development  from  system  development 

•  Calls  for  technology  maturity  to  TRL  6  (relevant  environment) 

•  Calls  for  evolutionary  approach  as  a  check  on  reqts. 

•  Short  development  cycle  times  (5  years  or  less) 


•  However, 

•  Best  practice  standard  is  TRL  7  (operational  environment) 

•  Most  individual  programs  do  not  even  abide  by  policy 

•  Many  programs  fall  outside:  satellites,  MDA,  ships 

•  Those  within  are  unique:  eg.,  FCS,  JSF 

•  Preference  is  still  for  revolutionary,  not  evolutionary 

•  Knowledge  gaps  and  optimistic  estimates  at  MS  B  are  the  norm 
and  are  reinforced  with  approval  and  funding 
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Prologue: 

DSB  Report  on  the  Effect  of  Immature  Technology 


"Immature  technologies  and  manufacturing  challenges  have  a  significant  impact  on 
DOD's  ability  to  rapidly  and  affordably  transition  technology  to  the  war  fighter," 

“SSI  program  managers  often  believe  that  affordability  and  manufacturing  issues  are  not 
relevant  concerns  in  6.3  programs,  focusing  instead  on  fabrication  of  test  and  evaluation 
and  prototype  articles.  But  this  line  of  thinking  leads  to  higher  costs  later  in  a  program, 
when  manufacturing  concerns  are  addressed  after  technical  designs  are  considered 
'ready/ :: 

“In  order  to  achieve  the  objective  of  lower  cost  equipment,  manufacturing  concerns  must 
be  addressed  earlier  in  the  program  life  cycle.  Production  and  support  costs  need  to 
become  a  component  of  key  technical  design  requirements,  before  the  final  stages  of 
development  when  technologies  are  released  for  prototyping." 

Delen&eGobi ns  &-wrd&IMh'  (date);  GAOftxeaentsdon  andfinsly&la). 
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Current  Outcomes:  %  Programs  that  Achieved 
Technology  Readiness  at  Key  Junctures 


100% 


75% 


50% 


25% 


67 

43 

10 

Development 

start 


DOD  design 
review 


Production 

decision 


Source:  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs.  GAO-06-391 .  Washington,  DC.:  March  2006. 


25 


i 


GAO 

Accountability  *  Integrity  *  Reliability 


Current  Condition:  Schedule  Growth  for  27 
Programs 


First  full  Latest  estimate 
estimate 


■  Months  (weighted  average) 


•  23.5%  change  in  weighted 
average  cycle  time. 


Programs  included  in  cost  and  schedule 
analysis:  AEHF,  MUOS,  NPOESS,  WGS, 
Patriot/M  EADS,  ARH,  Excalibur,  FCS,  Warrior 
UAS,  EA-18G,  EFSS,  V- 22,  AESA,  E-2D,  AHE, 
JTRS  HMS,  JTRS  GMR,  Land  Warriior,  WINT- 
T,  ERM,  CVN-21 ,  C-5AMP,  C-5  RERP,  F-22A 
Mod,  Global  Hawk,  JSF  Reaper,  P-8AMMA. 

We  limited  analysis  to  these  because  all  data 
including  cost,  schedule,  and  quantities  were 
available  for  comparison  purposes. 
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Private  Industry  Findings:  Adaptive  Cruise 
Control  Technology  and  Jaguar  Requirements 


Technology  Bridge  between  technology  Product 

feasibility  and  product  development 


A 

r 

_ 

A 

Advanced  vehicle  technology  office  ^ 

Jaguar  vehicle  team 

A 

A 

A 

Enabling  technologies 
pursued  in  the  technology 
base 

Interest  expressed 
by  vehicle  team 

Agreement  signed 
between  S&T  and 
vehicle  team 

Product  launch 

Model  introduced 

TRL  5 

TRL  7 

TRL  8 

1993 

1995 

1996 

1997 

1999 
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Private  Industry  Findings:  Use  of  Project  and 
Process  Metrics 


•  Project  metrics  used  to  assess  the  status  of  technology 
development  and  whether  the  technology  meets 
product  needs 

•  Size,  weight,  power,  and  reliability,  as  well  as 
nonrecurring  development  and/or  manufacturing  costs. 

•  Process  metrics  provide  information  on  the  status, 
timeliness  and  impact  of  technology  development 
efforts. 

•  Return  on  investment,  cycle  time,  technology  yield, 
number  of  technologies  commercialized,  customer 
survey  results. 
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Private  Industry  Findings:  Knowledge-Based 
Evolutionary  Product  Development 


KP  1 


Tech.  Development 

■ 

■ 

Integration 

Demonstration 

Produce 

KP  2 


KP  3 


Tech.  Development 


KP  1 


■ 

■ 

Integration 

Demonstration 

Produce 

KP  2 


KP  3 


Tech.  Development 


KP  1 


■ 

■ 

■ 

Integration 

Demonstration 

Produce 

KP  2 


KP  3 


S&T  Manager 


* 

Product  Launch 
Point 


Product  Manager 
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Private  Industry  Findings:  Notional  Boeing 
Technology  Maturity  Scorecard 


Technology  transition 
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DOD  Findings:  Use  of  Technology  Transition 
Agreements  Increasing 


•  Technology  transition  agreements  are  starting  to  be  used;  however, 
use  and  coverage  vary  greatly  among  service  S&T  programs. 

•  About  65  percent  of  technology  projects  identified  by  the  services  as 
transition  candidates  had  agreements.  This  is  a  small  portion  of  the 
total  S&T  project  portfolio. 

•  Agreements  contain  some  of  the  same  elements  seen  in  industry, 
such  as  technology  description,  key  personnel,  and  performance 
characteristics  required  by  the  prospective  user. 

•  Typically  do  not  require  the  technology  developer  to  demonstrate 
cost  metrics. 

•  Tool  used  mainly  for  S&T;  Acquisition  community  does  not  value  as 
highly. 
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DOD  Findings:  Metrics  Are  Underutilized 


•  Metrics  continue  to  be  acknowledged  as  a  problem  for  the 
services. 

•  Transition  agreements  include  many  of  the  same  metrics 
as  leading  companies,  but  are  not  available  for  most 
technology  projects. 

•  Few  metrics  are  used  to  gauge  the  impact  of  investments 
and  the  effectiveness  of  processes  for  developing  and 
transitioning  technologies. 

•  DOD  Foreign  Comparative  Testing  Program  has 
established  metrics  to  measure  health,  success,  and  cost- 
effectiveness  of  the  program. 
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DOD  Findings:  Use  of  Relationship  Managers 
is  Underutilized 


•  DARPA  uses  relationship  managers  in  a  manner  consistent 
with  leading  private  companies. 

•  When  used,  relationship  managers  generally  are  used  by  the 
military  to  market  lab  technology;  not  as  a  joint  communication 
vehicle  between  labs  and  acquisition  to  assist  in  technology 
transition. 

•  DOD  depends  on  Integrated  Product  Teams  and  annual 
corporate  board  reviews  to  communicate  technology 
development  activities. 
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Outcomes  Since  Then: 

Reduced  Buying  Power  For  Major  Programs 


Program 

Percentage 

Initial  Initial  Latest  Latest  of  unit 

estimate  quantity  estimate  quantity  cost  increase 

Joint 

Strike 

Fighter 

SI  96.  5  billion  2,866  aircraft  $223.3  billion  2,458  aircraft  32.8 

Future 

Combat 

Systems 

u*.— .  .  .  * 

taft  ^  ’*■ 

$85.5  billion  15  systems  $1 31 .7  billion  15  systems  54.1 

V-22  Joint 
Services 
Advanced 
Vertical  Lift 
Aircraft 

$36.9  billion  913  aircraft  $50.0  billion  458  aircraft  170.2 

Evolved 

Expendable 

Launch 

Vehicle 

TP 

$16.0  billion  181  vehicles  $28.6  billion  138  vehicles  134.7 

Space 

Based 

Infrared 
System  High 

< 

$4.2  billion  5  satellites  $10.4  billion  3  satellites  311.6 

Expeditionary 

Fighting 

Vehicle 

■SB 

$8.4  billion  1 ,025  vehicles  $1 1 .3  billion  1 ,025  vehicles  33.7 

Source:  GAO  analysis  of  DOD  data.  Images  sourced  in  their  respective  order:  JSF  Program  Office:  Program  Manager.  Future  Combat 
Systems  (BGT);  V-22  Joint  Program  Office:  (Left)  ©  2006  ILS/Lockheed  Martin,  (right)  ©  2003  The  Boeing  Company:  Lockheed  Martin 
Space  Systems  Company;  General  Dynamics  Land  Systems. 
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Outcomes  Since  Then: 

Same  Problems  Under  Revised  Policy  Circa  2006 


M  MA 
AESA 
JTRS  Cluster  1 
Global  Hawk 
Future  Combat  System 
F-35 
WIN-T 
E-2C  Hawkeye 
JTRS  Cluster  5 
Aerial  Common  Sensor 
EA-18G 

Expeditionary  Fighting  Vehicle 
MEADS 
C-130  AM  P 
C-5  RERP 
MP-RTIP 
SDB 
B-2  RM  P 


Percent  Cost  Growth 


Source:  Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost  And  Schedule  Problems  Under  DOD’s  Revised 
Policy.  GAO-06-368.  Apr.  13,  2006 
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